Randomization ensures that the groups being compared are balanced for known and unknown confounders.
Inadequate sequence generation can overestimate treatment effects by 12% (ROR 0.88, 95% CI 0.79, 0.99).[1]
ROB Decision
Criteria
Low
Description of a random component in sequence generation process (e.g., computer-generated random numbers, coin toss, random number table).
Unclear
Insufficient data is provided to make a judgment (e.g., only described as: random, randomly generated, randomized, etc.).
Statements that groups were blocked or stratified should NOT be considered sufficient for a ‘yes/low risk’ rating unless accompanied by a description indicating the sequence was computer generated.
High
No randomization or inappropriate randomization (e.g. alternation, assignment based on birth date or day of hospital admission).
Allocation concealment ensures that the randomization sequence is unknown to the person entering participants into a trial until allocation to an intervention has occurred.
Inadequate allocation concealment can exaggerate treatment effects by 18% (ROR 0.82, 95% CI 0.71, 0.94).[2]
ROB Decision
Criteria
Low
Sequentially numbered, opaque, sealed envelopes.
Sequentially numbered drug containers of identical appearance.
Central allocation with description (i.e. pharmacy, web-based, call-in).
Randomization sequence maintained by a 3rd party uninvolved in the investigation or off-site.
Central allocation, but no further description is provided. Use context to determine whether this better fits in ‘low’ or ‘unclear.’
Unclear
No statement.
Includes 1 or 2 of: opaque, sealed, sequentially numbered envelopes is described, but does not use ALL 3 descriptors.
Randomization sequence is kept on site (with no further description).
High
Open allocation schedule.
Any method in which the investigators could foresee the group assignment (e.g. alternation, date of birth, etc.).
Was knowledge of the allocated intervention adequately prevented during the study?
Blinding of key individuals in a trial can minimize performance and detection bias.
Studies not described as “double-blind” can overestimate treatment effects by 9% (ROR 0.91, 95% CI 0.83, 1.0).[3]
Results are not consistent across studies.
ROB Decision
Criteria
Low
No blinding or incomplete blinding, but the outcome is unlikely to be influenced by lack of blinding.
Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken.
Description of “double-blinding” with specification that the participants and study personnel are blinded.
Description of the use of “double-dummy” or “matched placebo.”
Unclear
Insufficient information to permit judgment.
The study did not address blinding.
High
No blinding or incomplete blinding and the outcome is likely to be influenced.
Blinding attempted, but likely that it could have been broken and the outcome is likely to be influenced.
Were incomplete outcome data adequately addressed?
Per protocol analyses may yield more favourable estimates compared to intention-to-treat (ITT) analyses.
“Modified” ITT* may overestimate treatment effects by 15% (ROR 0.85, 95% CI 0.81, 0.88).[4]
Results not consistent across studies.
ROB Decision
Criteria
Low ROB
ITT and dropout rate <10%.
No ITT, dropout rate <10%, and number and reasons for withdrawal balanced between groups.
Unclear ROB
ITT and/or dropout rate between 10 and 30%.
High ROB
Dropout rate ≥30%.
Dropout rate <30%, but number and reason for dropouts very imbalanced between groups.
Per protocol analysis only (unless under of patients switching groups is too small to make any important difference to the estimated intervention effect).
*Modified ITT is not consistently defined, but may refer to how participants contribute outcome data (e.g., all participants available for follow-up at timepoint X), and whether all or a subset of randomized participants are included in the analysis (e.g., excluding participants that did not receive a specified minimum amount of the intended intervention). In assessing this domain, closely consider the description of what was done, the dropout rate, and the likelihood for bias.
Is there an indication that the reports are free of selective outcome reporting (SOR)?
SOR occurs when a subset of the original variables recorded for investigation in the protocol is presented in the publication.
It may also be indicated by discrepancies in the primary outcomes proposed and those reported.
Statistically significant outcomes are more likely to be completely reported than non-significant outcomes (range of odds ratios 2.2 to 4.7).[5]
ROB Decision
Criteria
Low ROB
No discrepancies between outcomes listed in the protocol and those described in the results.
No discrepancies between methods section and results section in the study report(s).
Unclear ROB
N/A
High ROB
Outcomes are identified in the protocol or methods section that are not described in the results section of the report.
When to search for protocols:
Cochrane recommendations
Protocol should be compared with results.
In a previous study, we found that searching for protocols had little yield, but often changed assessment when found.
Protocol or registry number is reported
Look for the protocol and compare outcomes in protocol with the results.
Protocol registries: clinicaltrials.gov, ISRCTN, WHO, Australia/New Zealand registry
No protocol reported
Do not look for protocols.
Compare the methods section with the results for: primary vs. secondary outcomes; first outcome reported in methods vs. first outcome reported in results; time points of outcome assessment
Request guidance from clinical leads to determine if there are key outcomes that should be reported according to the report topic. Decisions on searching for protocols and how to use them should be made at the outset of each review, based on the specific context.
Source of funding or early stopping should not be assessed in “other sources” of bias; however, continue to extract these items and report them in the results/discussion sections.
Conduct a sensitivity analysis for studies that received industry funding versus no industry funding.
Conflict of interest can be assessed in the “publication bias” domain of GRADE.
Specific “other sources” of bias will not be identified to assess for each project.
ROB Decision
Criteria
Low ROB
No other sources of bias are identified.
This is the default assessment.
Unclear ROB
N/A
High ROB
Other sources of bias such as: participant characteristics (baseline imbalances), study design characteristics (crossover, cluster randomized, blocked randomization in unblinded trials).
[1]Als-Nielsen B, Gluud LL, Gluud C. Methodological quality and treatment effects in randomised trials: a review of six empirical studies. 12th Cochrane Colloquium Oct 2-6, Ottawa, Ontario, Canada, 2004 [2]Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457 [3]Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457 [4]Abraha I, Duca PG, Montedori A. Empirical evidence of bias: modified intention to treat analysis of randomised trials affects estimates of intervention efficacy. Z Evid Fortbild Qual Gesundhwes 2008; 102[Suppl VI]:9. [5]Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008; 3(80):e3081.
Download guidelines:
Table of Contents
Types of Biases and ROB domains
Allocation concealment
Sequence generation
Statements that groups were blocked or stratified should NOT be considered sufficient for a ‘yes/low risk’ rating unless accompanied by a description indicating the sequence was computer generated.
Allocation concealment
Sequentially numbered drug containers of identical appearance.
Central allocation with description (i.e. pharmacy, web-based, call-in).
Randomization sequence maintained by a 3rd party uninvolved in the investigation or off-site.
Central allocation, but no further description is provided. Use context to determine whether this better fits in ‘low’ or ‘unclear.’
Includes 1 or 2 of: opaque, sealed, sequentially numbered envelopes is described, but does not use ALL 3 descriptors.
Randomization sequence is kept on site (with no further description).
Any method in which the investigators could foresee the group assignment (e.g. alternation, date of birth, etc.).
Blinding
Blinding of participants and key study personnel ensured, and unlikely that the blinding could have been broken.
Description of “double-blinding” with specification that the participants and study personnel are blinded.
Description of the use of “double-dummy” or “matched placebo.”
The study did not address blinding.
Blinding attempted, but likely that it could have been broken and the outcome is likely to be influenced.
Incomplete outcome data
No ITT, dropout rate <10%, and number and reasons for withdrawal balanced between groups.
Dropout rate <30%, but number and reason for dropouts very imbalanced between groups.
Per protocol analysis only (unless under of patients switching groups is too small to make any important difference to the estimated intervention effect).
[back to top]
Selective outcome reporting
No discrepancies between methods section and results section in the study report(s).
When to search for protocols:
In a previous study, we found that searching for protocols had little yield, but often changed assessment when found.
Protocol registries: clinicaltrials.gov, ISRCTN, WHO, Australia/New Zealand registry
Compare the methods section with the results for: primary vs. secondary outcomes; first outcome reported in methods vs. first outcome reported in results; time points of outcome assessment
[back to top]
Other sources of bias
This is the default assessment.
Overall ROB assessment
[1]Als-Nielsen B, Gluud LL, Gluud C. Methodological quality and treatment effects in randomised trials: a review of six empirical studies. 12th Cochrane Colloquium Oct 2-6, Ottawa, Ontario, Canada, 2004
[2]Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457
[3]Pildal J, Hrobjartsson A, Jorgensen KJ, Hilden J, Altman DG, Gotzsche PC. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2004; 36(4):847-457
[4]Abraha I, Duca PG, Montedori A. Empirical evidence of bias: modified intention to treat analysis of randomised trials affects estimates of intervention efficacy. Z Evid Fortbild Qual Gesundhwes 2008; 102[Suppl VI]:9.
[5]Dwan K, Altman DG, Arnaiz JA, Bloom J, Chan AW, Cronin E, et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008; 3(80):e3081.
Return to: Tools